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ABSTRACT 

After reviewing the literature, the authors defined a 
two dim«5nsional framework comprising formative evaluation activity as 
one dimension and source of information as the other. Four types of 
formative evaluation activity were identified and defined. Three 
primary sources of information-internal, external, and 
contextual-were identified for consideration as the evaluator engages 
in the following four types of formative evaluation activity. The 
first section reviews a number of approaches to formative evaluation 
in the predevelopmental stage including sampling, Q-sort and task 
analysis. In the second section techniques for the formative 
evaluation of objectives are discussed, including questionnaires and 
surveys, delphi tpchnique, and content analysis of documents. The 
third section deals with techniques for formative interim evaluation 
which may include collecting internal information such as descriptive 
information and processing critical appraisals, as well as describing 
physical specifications of the product. The fourth section deals with 
formative product evaluation in which a version of the complete 
product is produced. Rather than being discrete, this stage is 
continuous with evaluation of interim stages of the product. 
(Author/RC) 



ERIC 



CDUCATiON « «%tLr ARC 
NAtlONAt iNltltUTr 0^ 
COUCAttON 



Mu-,M'lp MAtrniAL MAS H£IS OMASHO M» 



■> ■ 1 , 



' " '^^'>M V4 S • MA' Ml r S Wf '-I* ' 
A • S . • »" ' % ".fit . f ;^k, Ap. fc, • i,', 

r {in », > • sn I s A4. . r u t Kk| 



to tntC ANO OfClAN':AriON< (iftMAt'Su 



TECHNIQUES AND PROC KIHMUiS FOK FORMATIVi: I-VALUATION 



J am OS R. Sanders 



1 



and 



Donald J. Cunn''nghan 



Indiana University 



Sanders and Cunningham (1973) recently extended the 
writing of Scrivcn (1967) on the nature of formative evaluation 
appHcd particularly to tlie product development process. 
Formative evaluation was defined as the process of judging an 
entity, or its components, that could be revised in form, for 
the expro.<?sed purpose of providing feedback to persons directly 
Involved in the formation of the entity. The authors defined 
a two dimensional framework comprising formative evaluation 
activity as one dimension and source of information as the 
other. Four types of formative evaluation activity were 
identified and defined as fpllows: 

1. Pre-develoniaental Ac tivitics --formative evaluation 
work which occurs before formal product development has 
started. Formative evaluation tasks related to the 
evaluation of needs, tasks, other planning activities 
would fall into this caLcgory. 

2. Evaluation of Objectives Ac t ivi t i e s f o rma t ive 
evaluation work directed at judging objectives in 
product development. The emphasis of work falling into 
this category would be on the provision of reliable 
infer mati on about the worth of goal statements produced 
by the product developer. Both logical and empirical 
evaluation strategics were proposed. 
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^ • ro rt-Ki t: I V (! ^Jji.l'J:'Jljj'L- 'L\! ' ^ V) ^ ALL j ,f ? J V . ./V^ ' J ,LX J. , .1 / : j^. " " ^ ^ ^ ^ ^'^ 
work cIcaJinn v/iLl) I lio a p r :i i 1 of cnrly ]) rod net dovclopiacnL 
efforts, Koriiinl ovaluallon aclivitlos chaL would fall into 
this calpj»,ory were iiilcwJni payoff evaluation work, interim 
intrinsic evaluation work and tlio evaluation of pro^',rain 
or project operations. Informal evaluation activities > 
often unobtruuivoi were also discuSsSed, 

^* ' iL5J*Jl''i t i ve I'roduc t K v n :i u a 1 1 o n A c t i v i t: i e s - -formative 
evaluation work which focu«os on the appraisal of a 
finished draft of the propot'.cd product. Strategies such 
as validation studies, cost analyses, descriptive analyses 
and goal free evaluation directed toward a product draft 
would comprise this category. 
Three priniarv sources of information were identified for consider 
ation as the evaluator enf^a^^es in the four types of formative 
evaluation activity listed above. The three sources v/ere 
labeled and defined as follows: 

In t erna 1 In f o r ma t i on --in forma t ion that could be 
generated by inspect in^; tlie entity itself. Included 
in tliis category v;ould be descriptive information about 
and critical appraisals of the entity, 

2, Ex t ernal In f o rma t i o n -- in f or ma t ion concerning the 
effects of an entity on the behaviors of relevant groups. 
Student achievement after using a product or parental 
attitudes toward the objectives of a product v/ould be 
information placed in this category. 
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3 • Coiytyox r u a 1 F ii for i^V'i.L'jl \\~ " i n f o r in alio n c o n c o r u 1 n [; f h o 

'coiulilioiis uiulor which an cMitity Js oxpooLt»c1 to function. 

Clnsyroom env J r (uunu n t , pupil cha rnc tor J I icr, and I inio of 

year are throti o*ai!ip.lo.s of information that v;oulcl fall 

into this calcp.ory. 
The tv;o diiionsions were cros^icd in a .summary tabic that included 
evaluation techniques and proccdurey as cell entries. Tliat 
table is reproduced licrc as Table 1. 



Ttiblc 1 



Approprinic im- roriiulivo llvciluaLion 
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riu>i)i:vi:i.oriir.NTAL 


l'*VAl HATUi'J or 




.oiur.cTjyiT. 




logical nnn]>\^jct3 


lo,;,ical analyses: 


ma I c r i ;i 1 s analysis 


of nccdJJ: 


1, cogency 


guidel ines 




2 . ccnsequcinces 




2. conncciiicnccii 


3, hif;hcr order 


content analysis 


3. hlnhcr order 


values 




values 






empirical analyscis: 


ei::pirical atuilyses: 


analysis of learn- 


of need:;: 




ing structures 


lo j^roup data; 


1, group data: 




Liurvcyfi 


surveys 


group data (criti- 


sea] inj; 


scaling 


ca?, appraisal) 


Q-tecliiiiquc 


Q-techniquc 




seiuaiitic dif- 


seraantic dif- 


expert opinion 


f oronl ial 


fer cat ir.l 


(inducting author) 


Delphi tech- 


Delphi tech- 




nic lk: 


nique 


unobtrusive mea- 


sentence ecu-- 


sentence com- 


sures 


pletion 


pletion 




2 c observr^t ion & 


2o observation ^• 




expert opinion 


expert opinion 




unobtrusive 


unnbtru5: i vc 


PERT 


Tneasurcs 


measures 


PPBS 


accreditation 


accreditation 


system analysis 


procs. 


procSo 




category sys- 


category sys- 




tems 


tems 




rating EVbtems 


rating systems 




3o analyse is of 


3o analysis of 




documents 


documents 




unobtrusive 


unobtrusive 




ineasurcs 


measures 




content ana- 


content ana- 




lysiti 


lysis 






operationalization 


experimental and 


• 


of objectives 


quasi-experimental 






design 




experinental try- 






out of goal state- 


clinical methods 




ments 








quantitative natu- 






i\'ilistJc observa- 


• 




tion tcchniqi'cs 






"unobtrusive tnea- 






Gures 



rnonucT 

icost analyses 

matcria],s analysir 
guidelines 

content analysis 



group data (criti- 
cal app: aisal) 

expert opinion 



unobtrusive mea- 
sures 



experimental and 
qua s d - 'ii^: p e r im n t n 1 
desl ^n ; hypothesis 
testing 

cost analyses 

CFE 

correlational ana- 
lyses 

quantitntive natu*- 
ralistlc observa- 
tion t echniques 
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piu;-Di;VKLorMi',!;TAL 



rOj^MA'i" 1 \'! ; F. VAIUAVI ACT! VTTY 

EVAMiAvio:; or 
oi',.j; ;: n vi'.s 



H 
X 

w 

H 
>^ 
O 

o 



needs assessment 



contCN t ainsassmcnt. 
(ii no noccis 
assesr>irA;ni: result.s 
available) 



INT].:iriM 



literature ro- 
viev;s 



if! formal obcer- 
vation 



piionucT ^ 

unobtninivc mca/niro 

jjroup dat'a per- 
ce:ivc»d (on c fit 
tivcnoss of i)rouucl:) 

observation tech- 
nique 

ATI procedures 



contc::t aGSCcv".":\t 
(focuG on external 
validity) 
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The purposes of this JotuiiuMiU is to cxpnnd on the earlier 
work by t he nutliurs by c 1. al) o ra i: i n{; on j;olccLoc1 Lcchnlqucn and 
proccdurt's listed in the earlier work. Since r.pace limitations 
do not nl>ow the d ev o i opnie n (; of all tccliniques and procedures 
refcrenccul in the prevlouii v;ork, the authors have selected 
those techniques and procedures which appear to be usefu] for 
formative evaluation. In addition, the authors have provided 
references to relevant techniques and procedures that were not 
selected for elaboration. 



Tl-CIINIQUES AMD PROCEDURi:S FOR 
PRE-DEVK LOPMENTAL FORMATIVi: EVALUATION 



Procedures and techniques for pre-deveTopinenta]. formative 
evaluation are often non-existent in typical evaluation systems, 
or, at best, they are verv informal. Given the immediate need 
fo. production in most developmental projects, this situation 
is often explained away. But, it can never be reconciled when 
expensive errors are made during later stages of development. 
For this reason, ve recommend the fullest amount of pro- 
developmental formative evaluation possible (within the con- 
straints of scheduling, costs, and politics) using cheap 
approximations v?hencver formal, complete techniques and proce- 
dures are ruled unrealistic. The following arc a few of the 
methods that the formative ovaluator may want to draw on before 
d ev elopme n t actually begins. 

Reference was made in tlie earlier paper to needs assessment 
and needs (and object ) evaluation procedures. It is 
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liiH true I ivo in LhiH ropard to consider cnrcfiil.ly prcrccultircfi 
used by the National Ai; lu^ juuncn l: of KducaLional Proi^rouf; (NAKP) 
project and iho Institute for Social Kof.carch?- (ISK) at tho 
Univorriity of Michi^vni to idcMitify refined tocliniquof; for 
accompl ishinp, nocossary p r o-d c v e 1 opne n t a .1 activitios. The 
techniques described here arc ideals which can be quite 
expensive, but there is nothing to prevent the formative 
evaluator fron>. adapting ihem to meet his needs. Two technical 
problem areas that have emerged frequently are those of 
developing a good sampling frame and of planning data analysis 
androporting. 

The KAEP sajnpling plans have been developed to meet two 
criteria: high accuracy in p.irameter estimation and low cost. 
Tl r nation-wide probability sampling plan comprises a stratified 
multi-stage design. The parameter of interest is P^^, the pro- 
portion of the total number of persons in a certain subp opul a t ion 
of the United States that answers an exercise in a certain way 
(e.g. Pj^ - proportion ar.swering *yes*; P2 = proportion answering 
* no * ; P3 proportion ansv;er *I don't knov;* to a three-option 
exercise). Each parameter is estimated by first estimating 
the total number of persons in the subpopula tion , then 
estimating the number who would select each option on an 
exercise, and then expressing tlic estimate, P as a ration of 
the latter to the former. Such an estimate is called a * combined 
ratio cf.timate' when applied to a stratified sample. The 
sampling plan includes 1 i s t1 n g unit s (small geographic areas, 
often counties, \7ith a minimum size of 16 ,000 persons and easily 

ERIC 



ill cut if led bou nihn* ] I' s ) as the major unit:* V r 1 ii ki r y Hi^jiPj^l JlH 
Ujijj^; (l*i;U) are i t1 on t: i f 1. cmI within oach lijitinr. unit. In orilor 
to obtain a r.ainplo si;:c of 2,000 responses p(^r exercise (tlic 
n ncf.lod t:o obtain adcquaLc precision), 20S PSU's wilMi 10-12 
obr.o irvat tons on each ckcvcIso in each PSU used. Tho 208 PSU'b 
are di*:M...Mi from eaclj of the four major regions of the United 
States (52 from erih). PSU's arc st.ratified on size of community, 
accordinp, to iIjo census populations \s'ithin each ref;lon, on 
income, and on geo^^raphic location within a region. The sample 
is selected by first drc:\;int; 1-2 PSU's per stratum randomly 
witliout replacement. Then, for the in- school sample, SLudonts 
frcMn each PSU are listed. bometiracs large schools could 
contain students from several PSU's. Since a constraint of 
usin[; at least two' schools per PSU is placed on the sampling 
procedures, it has been important to associate students within 
the PSU with their schools. Approximately 250-350 students are 
expected to be within each PSU for each age group. Those 
students who actually participate in National Asse^^Mnent are 
randomly drawn from the 250^ 350 students in each a^^;'^ group. 
For the ou t-o f -school sample, PSU's are subdivided into secondary 
sampl i ng u n i t s (SSU) which are clusters of 35-40 housing units. 
This procedure is very similar to the ISR procedure described 
below. Ten SSU's are randomly drawn from eucli PSU and are 
expected to yield 12.5 adult respondents on the average. 

The Ins titu tc for Social Research also uses a multi-stage 
samp 1 i n i\ p 1 an f o r most of i t: s 1 a i* g c studies. The steps of the 
sampling proceaure progress through various stages of selection 



{joliii; from j;M'f,<'r to r.rialiof nroiis. Those nlcpu parallel 
clortoly tliost^ uf'.cil by NAi:i». Hriofly, I lu» ISU samp] Ln|; plan lead 
to tlic J (I (Ml L i r i c a I ion oT Hjlij'iajlZ .^L'JillC.l liLll iLlLL^^ii (I'SU), uiiualJy 
count! or ric i r f>pc> 1 J t a ii areas, first, Tlu* I'SU's arc thon 
strati fie- cl by rol. ovant; d i ricnp 1 ons , r.uch as urban vcm'gus rural 
areas, incoino of arc.-js, and so on. A total of 7 A PSU's arc 
tbcMi raiuionly dravni from tho strata, proportionate to the total 
nur.ihei' of PSU's in each stratum. The selected TSU's arc then 
subdivided Into siualler areas called sample plac e s and each 
sanple place is subdivided into chunk s which are areas within a 
sample ])lacc which liavc identifiable boundaries (c.n« township, 
city block, an area bounded by identifiable roads, streams, etc. 
Several chunks are tlion selected randomly from each sample 
place for the sample and d\; e l ] inf . un i t s are tlien identified 
within the selected chunks. It is here that the ISR procedures 
begin to differ from those of NAEP, tlie reason being the 
different purposes of data collection for the two projects. 
A final step in the ISR sampling procedure is the random 
selection of 3- A dve 11 i np. units within a chunk , called a 
s egmen t , and these dv;elling units are used in the study. 

The value of using multi-stage scientific sampling 
procedures for collecting survey data should not be under- 
estimated. Obtaining precise estimates of relevant human 
paramc^ters is an essential part of quality (precise?) product 
d eve 1 o pra en t . V/ljilo the above discussion has not been 
prescriptive by aiiy moans (considering', the almost infinite 
number of variations of basic multi-^stage sampling plans, 
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c oil t c>; I * f r (• o r oi:)'Ui lul a L 1 o nr* arc v i r t ii a 1 ly :i inpo r. i 1) Iv ) , wc 
would arj;iii* for ttu» ailaptail^n of junnplinn planjt ali'cady in 
uii o for i> r c-d vc lo pr. i a 1 f o r ija I iv e c va Tuo t ion • 

Rc'porLlnj^ proccHlurcs f(>r luirvoy data usod by NAKP provide 
n Uiicfu] model for reporting; pr c -d e v (» ] o pruMi t a 1 fornativc c^vnlnnti 
activitirj-* It ir most* in : o r i la l: i v c to the rcadcM* to provide 
lhi» conplclo item alunp, with i^stiniatos of the proportion of 
persons in each r> ii h ju' p u 1 a I i o n who would choo^;e each item. It ifi 
also Inipnriant to provide nornative data for the siibpopul a I i on 
whieh can be used Lo inLerprei the reported parameter estimates. 
For exariplc, data reported by eo [;r a ph i e a 1 rc^^ion on nn overlap 
item given to respond ctnts ap.ed 13, 17 and adiilL, should include 
roi^ponsi^ cr: 1 ir;.a t es f^jr the entire popul::li on (over all ^eo<^ra- 
pliical rerjons) an well as data by aj^e reported side by side 
for comparison purposes, Tt is recommended that all planninf, 
data (on needs, objectives, etc.) be reported alonjf^side data 
on relevant referent j;ronp?i (norn^s). Descriptive statistics 
and parameter estimates (alon*; v,'ith standard errors) are the 
most uscfn] data reduction procedures to use at this staf',e of 
the development of large-scale assessment procedures. 

One technique for evaluating needs and objectives frequently 
mentioned in many recent papers on formative evaluation is the 
Q s o r t . Methods for collect Jtij; appraisal or judp.inental data 
from relevant groups of persons on simply and tersely stalc»d 
needs or objective!; if; essential in pri»-developmen tn 1 and 
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cl\i i'l' L I Vim; I (n i m I I v t- c v .1 I ii .1 1 1 o n * 1*0 r t tin.j I 1 y , llu' proccduriMi 
d ov 1 1 of, imI liy \.'MJi,uii S I 4' iMi so n ;nu! lalu-lcil (j iic I 1muK>J oi;y an* 
moiJl ;ippropr i ;i I c . The Q Ic cIi n J (pu' 1 j; Uio lofMcnl i» lu- r .1 1 J on ;i ] - 
l^'.aLlcMi of S I I'p li m >: on * s i )ifo r c L 1 r.i 1 Q ruM h odo ] nj»y . lH*Icfly, 
a 1 1 i; I 1^ f ncud I h t on?cn i fi or y,o<i\ (objrctivo) nlatt'runlM may 
bv. ii j.:^ J nctl nu:.n! r a 1 ?; , plncvt! on cards and j;Ivi;n to purnoiii^ tu 
rani; order acocn-dinp to soru* proilc I tTmincd ruJt»fj< The ordinal 
data lliat- rcriili fron lln' r,ortr. ruiy then ho analyr.rd Lo yield 
a nti rubor of iisoful slal Isl icr; hucIi a:3 : 

!• Cctui;lMvncy or Iujho j'.cn c i t y of rani; in^; \;ithin a i^voup 
of porLons ( n svc r i n f, t tie qiur. lion of how much do ]ieoplc 
n>;rer on thoir p c r c c i> I 1 onr. of the nccdj^ or ob j cc i. i von ) . 

Overall (and 5uj1» i* r 011 pcd ) rankings (or scLr* of 
prioritie;) on the list of needs or 0I1 j er* t i v e r. (and also 
the variance for each need or ol\jeclive t a l omen I ) • 
3, 1)1 f Ceri'nces in lankinji; profiles ainonj; i; roups of 
porsoni; (e.;;. a iimi;:ary of dlfferencen anronj^ n ychool 
boai'd, the rcbool ti^achers^ I h<» school a d n i n 1 s t r a to r s and 
parents on the p r i o r M: < o s or values a f ; s f |; n d to a list 
of needs c> r o t) J e c 1 1 v c» r; ) • 

A, Cluster*; <i f needs or objectives as ranked by a j;iven 
(;rou p of pi' r son s • 

5. Clusters of pcrr.ons as they rank needs or objectives 
(f*'^' Do Kt'publ leans versus neinocrals* clur..<»r r (*s]) ee t J v e I y 
on the i r priorities?), 

6. r« i tn J I ;t J' i I y 0I the il j s I r J !mi I J on oi lanl'lnr.r. by a ,;roup 
of persons tu an jdual ur criterion d i s t r i b u t ioa • 

ERIC 



- ,12 - 

There ari' two bcif;Ic t ypcji ci f Q-f4orl , onch wJl h n parLlrular usf j 

i*A^l!V'J.,M.C^'iL '^^^^^ iUl'LLliiiiJJV.^'^l* S t r lu: I II V ovl 0-.s(n*ln are those 
that Jncliule u nrt of rvil(*i? where hy a certain number ol cardn 
(needs or objoctJvtrs) must be placeJ in each of a certain number 
of piles (e.g. left-band pJTcs for most valuable and rlpbt-band 
piles for Jeiir. I valunbli! mcds or objectives). Here v;o arc 
forcing; the sort into a ]) r ed t» t c rm i n e d distribution, accordinc 
to some tlioory. Unstructured Q-sorts are those used whore 
there is no under] ylnp, theory and we ask a person to merely 
place the cards inlo a p r t; d i: t c vn in ed number of piles according; 
to his own perceptions of wliere they sliould be placed. In 
essence, we are saying here, "Let the cards fall wher(j they may. 

The procedurer.. used to collect Q--sort data generally 
follow tliese steps: 

1. Place unambiguous needs or objectivro statements on 
cards, one to a card. Theoretically, at least 75 but no 
more than lAO items should be sorted. 

2. Shuffle or randomly order the cards and give them 
to a person to sort. Th e same random order s ho u 1 d b e 
given to each person. 

3. Sort the cards into some predetermined distribution. 
Usually 7-13 piles of cards are used, but this can be 
modified, depending on the needs of the investigator. 
For example, if 80 items vero to bo sorted into a quasi- 
normal distribution, the followinp^ rules mij^ht be set: 
Sort the cards into 9 piJes with tlie n\ni\ber ir eacli pile 
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t< el ii s f 1 1 : 

A 6 ] 0 1 :! 1 G 12 10 6 A 
Tho Icfl-niosl i 1 o r c p r (> j; c n L 5^ nio«l valuable luicds or 
objectives iMul the r IrIi t-iiioj; I: pile i-cprcscnts least 
valuable uc e d r o r o b j tu: t i v o s . 

A. (lollect th(? cartlr> as sortcul by the person and asnljvi 
ranks to the cards in each pile (c.n« oi'^c to cards in 
left-most pile and 10 to cards in rii^ht-iaost pile). 
5 . Calculate d e s i r vul statistics on resultant d a t a . 
This technique has been used by numorou:; formative evaluators 
in Ko£;ional /Labs and II L D centers, schoolri and universities 
for collecting judgmental data necessary in planning for 
product development . 



A set of procedures which were not listed in the earlier 
paper by Sanders and Cunningham, but which should be essential 
to systematic product development are those that fall under the 
rubric of ta sk analy s is * This activity has been described in 
excellent discussions by Davies (1972) and Thiagarajan, Semmel 
and Semmel (1973). Early work by developers of programmed 
instruction has also contributed greatly to the refinement of 
task analysis techniques. This activity is not clearly 
evaluative since the judging process is not involved, but task 
analysis (like objectives writing) is often a function assigned 
to tlie cvaluator by hi:^ clients. 

There is some uncertainty about wliethcr this activity is 
appropriate at the p r e- d e v e lo pmcn ta 1 stage of product development 
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(wc arp.uo lat:or thai i l: iy d o T J n i I: a ] y an appropriate interim 
formative pvaluaLlon activity). Wo have Includotl a difJCUMslon 
of task anaJysir, liorc» because \7c feci l.haL at a point when 
Rlobal "needs" and "r>oals" are tlic only existing, d o s c r Ip t io n .s 
of the final product » it is worth partitioning these global 
outcomes into component parts. This activity could then lec?d 
directly into the preparation of interim and terminal objectives 
w h i c h have c u ^; e n t b a .s g s for their e >: i s t C' n c! e • Because of the 
uncertainty associated with the appropr la tener;s of task analysis 
activities at this point, we have discussed the teclinlque both 
here and in the formative interim evaluation section of this 
(jL'/p Vr • Task analysis techniques are Involved primarily with 
the prescription of the prerequisites and conditions under which 
behaviors raay be developed and a description of the behaviors 
which comprise a given performance, Thiayarajan, Semmel and 
Semmel list the following ste])S for performing a task analysis: 

1, Specify the main task [or performance]. This 
statement should indicate what the jubject is to do upon 
the use of a given product and the situation in wh i c h 

he is to perform, 

2, Identify subtasks. These statements should include 
the skills that the subject must possess in order to 
demonstrate the criterion performance, 

3, For each subtask, identify sub-subtasks which 
contribute to that subtask, 

4, Terminate reduction of tasks into subtasks when 

the subtasks are equivalent to the subject's entry behavior. 
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An oxnmplo of such c\n analysis i r, found in rijjurc 1 



Rcailjj ora].ly wordy 
conforming, to rer,u].ar 
p 1- o m I n c i. a 1 1 o n i u 1 o « 



Tects cue 51 to match 
syllables; to those 
familiar in oral 
vocabulary 



PronouncGc; total printed 

\;ords coKipocicd of 
Gcqucnces of consonant- 
vo v;*:: 1 c onib in a t i on s 
according to rcf;;ular 
rules 



r 



Pronounces regular 
'^spcllir.f, patterns'^ 
involvxT'.s samo vov/Cils 
in different phonemic 
va 1 u e G (ma t -n^.a t c ) 



1 



Pronounces two- 

and three-letter 
vowel-consonant 
con^binations 
("blending^') 



Pronounces 
cinclc vov.-els, 
v;ith alternate 
phonemic values 



J. 



Pronounces single 
consonants and 
diphthon^/,s, with 

alternate phonemic 
values 

- - ..'JjC 



Identifies printed 
letters* by sound 

T 









Reproduces orally 
presented words 
and word sounds 

of several syllables 
in length 




N 


1 




T 

Reproduces oralJ.y 
presented single 
syllables 




S 



Reproduces sinj'J e letter sounds 
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FIGURE 1. A learning; hierarchy for 
(•Mecodinr/') . 



a b a s i. c re a d i n g s k i 1 1 
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Davics r.ut;j;eu^ t cd that* task nnnlyrjis approaches jnich nr. the 
one ' d e c r ib o(l above avi* hul one of six dlCCorcnt approacl\es. 
The i>; c.atof,or i.ci3 of tar, k anal y sir; arc: 

!• Task analysis based upon objectives. This method 
includes the specification of Instructional objectives 
and the specification, for each objective, of the type 
of behavior (e.g. knov/lcdge, comprehension, receiving, 
responding, etc.) required for each. 

2. Task analysis based upon behavioral analysis (above). 

3. Task analysis based on information processing. This 
metl'iod includes a prescription of information to be 
processed for the performance to be mastered. Consider- 
ations of cues, manipulations to be made, feedback, etc, 
are central to this method. 

4. Task analysis based on a decision paradigm Underlying 
decisions which must be made to perform a given task are 
analyzed and decision chains and procedures are provided. 

5. Task analysis basod on content structure. Th.is 
method includes the identification of rules and c>;ample8 
invalued in the task, the presentation of these rules and 
examples and the discussion of relationships between them. 

6. Task analysis based on vocational schemes. This 
method involves the reduction of a performance into jobs, 
dutJ.cs, tasks and task elements. 

Davies noted that these six categories are not mutually exclusive, 
but do suggest central elements of different approaches to the 
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proMiMu of task an«'i 1 y i s . Tlu^ iilate of (:ho art: Jn t.lrir> ci.ri'a 
in fuich thai a c.o n r. :i d or ab 1 c aincuinL of rcMioarcli :l still ncuulccl 
to expose ttic uti]:ity of eacli of task analysis approach for use 
in qualiLy product d ovolopmen t . Wlii 1 c '''^ a p p r o a ch e s are quite 
proi'nsin^;, the mo. Liioc!.s of task analysis are still evolving. 
Tlu;y should he refined by formatJ.ve evaluators and data should 
be presented on the relative payoff of each. 

While task analysis and learner analysis are not clearly 
evaluative functions (for the same reasons that needs assessment 
and objective's en o r: t :i o n activities arc not evaluative), they 
are evaluation-related functions that are essential parts of 
the development process. Often formative evaluators are called 
upon to perform sucli functions, and, as such, they should be 
techniques and procedures which the formative evaluator has in 
his repertoire. The techniques and procedures used to evaluate 
needs or objectives (as outlined by the authors in the earlier 
paper) are appropriate for the evaluation of task analysis and 
learner analysis r e s u 1 1 s . 

ADDITIONAL KEFERKNCl- S : -'^^ Surveys (Herriott, 1969 ; Institute 
for Social Research » 1969; Oppenhcim, 1965), Scaling (Torgerson, 
1958), Q-teclinique (Stephenson, 1953 ), Semantic differential 
(Osgood, Sucl, and Tannenbaum, 1957), d el ph i t cc hn ique (Helmer, 
1967), accreditation procedures (NSSK, 1969, 1970), observation 
techniques and category systems (Simon and Ro^er , 1970; VJolcott, 
1968; Kurnett, 1968, 1969), rating systems (Lav;son , 1973 ),^ 
content analysis (lierelson, 1952; Guttentag, 1971). 
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TKCIIK'IQUKS AMD PUOCl'lDllK !• S FOR 
TlIK FORMATTVK KVAl.lJATTON OF OHJl^CTlVl-S 

The Importance of empirical mc(:l)oJs in evaluating objective?) 
was noted in the earlier i»aper by Sanders and Cunnin £;ham • Stake 
( 1970) suggested tlirce categories of judgmental d;ita that might 
be collected in evaluating objectives: group data, expert 
opinion and observation data, and document analysis data. 
Two essential strategies for collecting group data include the 
use of survey questionnaires and the delplii technique. Student 
opinions and content analyses provide valuable information from 
"experts" and documents respectively. 

The development of questionnaires is the most critical 
and possibly the most und er empli a s i.-^ ed part of survey inquiry. 
A common attitude among many evaluators and researchers is the 
predisposition to V7rite quickly a list of questions to be 
answered, put them on a form and call the resulting instrument 
a developed questionnaire. In reality, if the questionnaire 
has not undergone critical appraisal before being sent to 
potential respondents, little usable information will be 
yielded. The early stages of survey design comprise decision- 
making about the aims of the study and identification of 
hypotheses to be tested or questions to be ansx^^ered. Talking 
to experts and reviewing literature related to the evaluation 
focus should enable the evaluator to get a feel for the problem. 
After deciding on the questions to bo ansv/cred, it is important 
to consider the analyses, results, etc. needed to ansv/er the 
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qucn t ion^.J . At Llwit point:, flic cvaluiiLor shoulJ bo nblc l:u info, 
the . qucL; llonnai r e (jucslions that arc to ho ask(^d and 1k)W tlicy 
should hc» quantified. Criteria that could be used to evaluate 
draft versions of the que s t: i o n n a ir e include: 



1. 



C. 



Are la (: 
Is the 
Does it 
Arc lea 
e f f i c :l e 
genera 1 
q u e s (: i. o 
Arc ope 
If clos 
e X c 1 'J 5: i 
collec t 
Are the 
minor i 
Are c{ u e 
1 o g J. c a ]. 



fj u c n_ 
i.^ 1- r 
q u e s 

s t a 
din[; 
n t s 

to 
n s v; 
11 /cl 
ed, 
V e ? 
ed a 

ma j 
s s u e 
£5 1 io 
ly? 



c o 

e s \> e n s c 
t io n n a i 
rt off 

q u e r> t i 
e q u e n c :i. 
5 p c i f. 1 
hen app 
o s e d en 
are the 

(Gov 



s i. n t e r 
o r i B s u 
s a r e p 
n s V i t h 



b i a s C! d 
r e a 1 1 r a c 
\j i t h e a y y 
o n s a 5) k e d 
n g of c[ u e 
c que s t J. o 
r o p r i a t e ) 
d ed qu a:^ t 
c a t e g o r i 
ordinal 
V a 1 d a t a ? 
es cover 
a s s ed ov e 
s i m i ]. a r 



by early question s ? 
t i V e and i n t e r e g t i n g ? 
, impersonal q u e s t i o n s ? 
? Is there a logical, 
s t ionr. (e.g. f r om 
n s ; use of filter 

7 

ions a p ]:> 1" o p r i a t e ? 

e s c X h a u s t i V e > mutually 

or r. 0 1:: in a 1 data be 

) 

d thoroughly while 
r qu ic I'vl y ? 
content grouped 



II 



Q u est i on \-o r d i. n 



(\U\ o , wha t , when , 



A . Are u c s t i o n s stated precisely? 

where , wliy , hov; ? ) 
B. Does the questionnaire assume too much knowledge 

on the p a r t of the r e s p o ii dent? 
C . Are d D u ]:> J. e cj u c s L '.i o n s asked? 

D . Is the r e s p o n d cm t in a position to answer the 

qu( tion, or must he mal;e guesses? 
E . Are d ( • f :i a .i 1 1 o i i s c: 1 c a r ? 
F . Are e v.\ o t: i o n a 1 1 y t i n g e d \'; o r d s u s e d ? 
G. Ave tccluiical IcrnK';, ■javp. on, slang, words wllh 

d o u b 1 o m c a n i n s a v o :i d c» d ? 
H . Are t h o \:\ c f 1 1 o d r. f o v r e y. p o n ci :i. n g c o n i n I c n t: ? 
I . Are the ques tions impersonal ? 
J. Are the questivins short? 



III. Es tab 1 ish i n g and Ke ep i n g Rap p or t 



A. Is the questionnaire easy to answer? 

B. Is little respondent time involved? 

C. Does the questionnaire look attractive? (e.g. 
lay-out, quality of paper, etc.) 

D. * Is there a 'res])ondent orientation?' 

K. Is the questionnaire introduced v;:ith an explanation 
of p u }• p o s e • sponsor s hip. method of r c s p o n d c r. t 
select i on, a n o n y rn i t: y ? 
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A. Is rcispoiul on I: clearly told liow to record hiv* 
reapons oi; ? 

B. Are ins t: rue tie ns for return duo date nnd procedures 
in c lu d C'fl ? 

V • Tcchni cal On j 1 iCy 
A . Valid J t;y 

1 . A r second infer in a t i o n s o u r c e s u s e d n s 

c r or-: s - c 1) ecks ( ;i. n t e rv iev;e r r a tings, other 
findings, etc.)l 

2 . Are responses of like respondents (e.g. 
h u s b a n d / v; i f c ) check e d ? 

3 . H a V c o 11 1 e n t e x pert s r cad ]) i 1 o t vers i. o n s 
of I h c q u e s t J. o n n :i ires? 

B . Reliahilxiy 

1 . Are factual questions r ea sk ed ? 
2 . A r e p l i o n c y 1 1: e ii; s used? 

3. Are respondents r c in t erv 3 cv; ed ? 

A. Have res])onscs been checked for logical 
c o n s 1 s t e n c y ? 

C. External Validity 

1 . Are n o n - r e s p o n s e bias checks planned? 

An excellent annotated b ib 1 io ^^raphy on the design, construction 
and use of questionnaires for Inquiry is provided by Potter, 
ct al. (19 7 2 ) . 



A variant of survey procedures for collecting judgmental 
vl<iia is the delphl technique. This teclmique makes use of a 
panel of exj^erts v7ho are mai.led a set of questions to which 
they respond independently. A follow-up questionnaire reports 
a summary of the origina]! responses using the median and inter- 
quartile range as descriptive statistics for the responses to 
each original question. " Each panel member is then asked to 
reconsider his first responses and x^evise them if he so desires 
If his second response i.s outside tfie interquartile range, he 



is asked to justify his deviation from the majority judgment. 
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In the third round, tho ru.H'.ond round roiipon?;os arc summarix.ed 
a n d • a i\ u mm n r y of c h a r o a o n s p r o v ;l d d f o r d o v :i a n t p o v, i lion s 
is also included, Isach ])anol mo nib or is asked to r ocon?.; id cr 
his second round responses ivcn the results and reasons 
yielded from that round. A respondent who desires to remain 
outside the in t or quar ti]. o range on the third round J.s asked ♦ro 
present his reasons. This iterative procedure can continue 
for several more rounds after the third, but the payoff begins 
to diminish quickly. On the final round, panel members are 
asked to revise Llieir re5q'»onses one last time given the results 
and arguments yielded by the previous round. This procedure 
has been used in management to attain consensus judgments from 
a panel of experts. Often 'the results have been less than 
spectacular due to wealiiiesses inherent in tlie process, out on 
many occasions useful results have been obtained. This is i\ 
procedure that the formative evaluator may find useful in the 
early stages of product development when committments on selected 
developmental goals must be made. 

Abedor (197X) suggested procedures for collectinf^ judgmental 
data about objectives from representatives of the target popu- 
lation for a product. His procedures could be adapted so that 
subjects are given an objective oy( a list of objectives and are 
asked to react to them as behaviors that the subjects could he 
asked to demonstrate after usin^^ the product. A valuable 
lesson here is that members of tlie target population are 
'experts' v;ho are often overlooked in formative evaluation. 
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In ciddition, Mioy arc one of the most critlccil a lul i n s i lit: f u 1 
audiences available to the evalualor. Since they will be 
sufCerin^>; the con r; c q u en c e g of bad development: in the lonj3 run, 
they have some t:\i i n to lose by not providing feedback to the 
evalua tor. 



For the analysis of docunents for collecting judgmental 
data about objectives, content analysis procedures have much 
to offer. Content analysis aims primarily at th(^ objective 
quantification of content classified using a system of categories 
and explicitly formulated rules. The categories should be 
developed to fit the questions to be nnsv/ered by the data and 
they should be mutually exclusive and exhaustive. C o d i n u_nj-jt^ 
(e.g. words, tliemes, paragraphs, etc.) are what the content 
analyst actually counts and places within the categories. A 
sample set of categories into which themes contained in newspaper 
articles dealing with sex educat3.on could be tabulated might bc:^ 



Newi-paper 



Date : 



Story Source 
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Expressions of opposition 
to sex education 




Expr es s i on G favoring sex 
educa tion 




Actions in opposition to 
sex education 




Actions; in s u p ]•) 0 r t of 

s e x e. d u c a t i 0 n 




Statements s u p p o r t i n (7. 
opponents of sex education 




Statements attaching oppo- 
n en t s of sex e d u c: a t i 0 n 




Statements attaching pro- 
ponents of sex education 
Statements listing oppo- 
nents of sex education 
P r 0 V i s i 0 n r. of :i 3 t c ma t e 
p 1 a u s 




Statemoits supporting pi'o- 
p on e n t s of sex c d u c aj: ion 
Statements list in g propo- 
nents of sex education 
S t a t cm V n t s 0 p p 0 s i n g 
a 1 1 e r i\ a i c ]) 1 a n s 




Some other plan satis- 
factory 




A u t h 0 r i 1 1 c s i n s i s t 0 n 

c u r re n t, p b. j.e c t i. y 0 s 




M i s c e 1 1 a n CO u s - 


— < — 


M i s eel 1 a n c 0 u s + 



















- 23 - 



0 



0 I ho r thomofi 



Po ss^.L b ] o A r (.'t'l of C oil) J) r oi:i L o* 
M:i. fJ c e .1 .1 :in i.'o u s " - . . 



T ^LAUh? 

School lioiird Vole lo be 
clcisa _ 




Con ten I total s 



Head 1 X n c 



Headline Content 



H cad S i e 



(M, -1, or 0) 



Location on P a o 
Lcnp, th 



T o t a 1 Score a n d 



D.lr cc 1 1 on 



The uses of this technique for collecting j ud grnon t al data on 
objectives are iiumy. Thematic analyses of board mectinp^.s or 
editorials in pr o f c 8 53 iona 1 journals or word counts on federal 
policy statements can identify and clarify value data tlii^t are 
unavailable from any other source. 



At thifj point in the product development process, "pieces" 
of the intended final product are beginning to emerge. A film 
maker, for instance, often begins by constructing a series of 
verbal descriptions of the visual stimuli whicli he intends to 



of the stimuli to be presented on the sound track (if any). A 
frequent next step is the construction of a "story board," or 
simulation of tlie visual and oral stimuli with liand drawn pic-" 
tures or photograplis serving for the vii3ual stimuli. Some 



TECHNIQUES AND PROCEDURES FOR 
FORMATIVE INTERIM EVALUATION 



film, CO or d ina t i ig that description v;i t h a preliminary version 
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film producers iiuc the rcO a lively I (Mis cxp cmk*; I v o v id c o t .'i p c 
mod*] Hi.; lu filiii In ilia] verrJniiM of (lioJr film for d cM) ii li^^", in 
pur pesos. Film i.s a particularly difficult modi urn to revise 
once it ha« reached the fin Is lied product stn[;;e in that changes 
are likely to cor.l as much a« the product itself. So it is 
extremely import.int to locate points in the interim stnj^es in 
development of fi]:::cd materials where evaluative information 
can be provided concerning; potentially useful revisions. 

Hut, the fact that revisions of a finished product are 
less costly in some ithcr media should noL obscure the usefulness 
of seeking evaluation at the interim stage of product develop- 
ment. Most textbook authors begin by constructing some sort of 
topical outline, chapter summary, etc. Rough drafts of chapters 
often undergo several revisions based upon feedback froia 
colleagues, students, spouses, secretaries and anyone else who 
the author can coerce into reading his smudged drafts. Small 
scale tryouts of each chapter as it rolls off the pencil are 
often undertaken. The point is, of course, that in the develop- 
ment of nearly any product many opportunities exist prior to 
the completion of the initially satisfying version of the 
complete product for evaluative information to be collected. 
The particular techniques useful for formative interim evalu- 
ation of various media will differ somewhat from medium to 
medium but many general principles can be noted. 

Formative interim evaluation information can involve 
collecting internal information such as der»cr :i pt ive information 
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and prtu cs s i np, o r i I i r a I ;i p ;> r a i mx J j. • 1)c5J c r i [> t i vi.* Info rM.i t \ r» n 
Vi^fciN to the ot')('cMivr i u 1* T Ji a L i (M) \;hlc'li cni hv [\rnmMv<\ hy 
iUHpcrLin); the ylvccj or prtlJMlntiry vcrsiouj.; of ihc j)r()(huM:. 
Crilli:nl npp r;i i .sa Ifi nro J ud ^.tu-n I u iisndc concirnlnij llu! plccct; 
by ropr or. cnia t i vc'i; of concerned popn la t i on s (c?,^;, cxju?rla, 
parcMiLf. , student fj, etc.)- Kacli of IhofJo wili be disctisncd In 
turn . 

Thi* Intent ot collevilnp. d c r;e r i 1 v e infornatlon is to 
describe fully .'\nd comp]elely what jji^ not v.»Iuit should be, 
A coi,:proluMi5 LV(! c ba r ac I e r i ;'.a t i 0 n w'pat 1g will aid prctotly 
in laakinr; j ud ;;;:jon t and in d e t e r m 1 n 1 n where to rcvlr.c once 
some deficit is identified, 

One type of descriptive in f ori;ia t ion is ))liy s 1 ca 1 iipc*£-J LL*" 
cations which iii r> Imply a description of the primary "tangible 
clui r a c t er is 1 1 c .s of the prodiict consist inj; in larpo part of 
media characteristics. This type of Information is best 
collected by ruianfi of a checklist wliif.h inclndor. the majority 
of the characteristics upon v;bicb products can vary. Tlieso 
char ;;c I er i r t i c n arc ununlly ncdin r.pecific in that any f'cneral 
purpose checklist would be impoyr.ibla to construct. Some 
sample clia ra c t e r 1 s t i c s nro listed in Table X usiny programmed 
instruction as an illustrative medium. 



Tab To 2 



Snmp I c 1 1 ens f r oin n Ch cck .1. J s t for KVi-i lua I in ^; 
Dc.Hcr Lp live Ch a r a c t: cr 1 s t i c {J of a Proj;raminecl Textbook 

Pre- test pi'(»vlclccl? Vcs No 

Objectives listed? Yes No 

Confirmation proceduro. Check one, 

Know1.i'dj:o of roi^ultfi provided on sane pap,e> students 

asked I o s h i e J. d answer. 

Knov;.l fd 5',e of rer>ult5 on another page of text. 

Knowled[',e of results provided in separate booklet. 

Knowledf.c of results not provided - 

Otlier (Please specify). 

Response requirement (intended) : 

Over t cons true ted . 

Covert constructed. 

Overt selection. 

Covert selection. 

Other (Please specify). 

Can student alter response requirement? Yes 

Blackout Ratio. 

% of material could be blacked out. 
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Some po.i.ul:s In the nan'^^le checklist note whether certain 
features are presenl; or abfjent in this product. (See j)Olntr» 
one and two.) OLher points identify the type of procedure that 
is employed for featuroirJ v;hicli arc preseJit invariably (or nearly 
so) for every product of this type. That is, nearly every 
program requires some type of response from the student and 
nearly every one provides knowled^;e of results in some form. 
But different programs require different types of responses and 
use different methods of providing knowledge of results. (See 
points three and four.) 

Similar checklists could be developed for any medium or 
combination of media. Checklists have, of course, been developed 
and used for many years (e.g. Edmonson e^ iLL» » 1931; lloban, 1942) 
but these checklists require rather global judgments by the user 
and are likely of more use to the summative evaluator. One 
would expect that a number of generally accepted checklists of 
potentially useful descriptive information for formative evalu- 
ations would be available for instructurrl products of many 
types but, to our knowledge, this is not the case. Each 
developer, if he concerns himself at all with descriptive 
information, rediscovers the wheel, so to speak. The disadvantage 
of such a state of affairs is that t])e developer may not be a:;are 
of potentially useful types of descriptive information. 

One method wliicli offers promise for describing product 
content is content analysis. The content analysis procedures 
discussed earlier in this ^Ktii-er are also appropriate for formative 
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intrcrlm evnliui lion. I3cro.ls;on defined the technique as 

a "foJ^cai'ch toclniiquo for the objective synlcmatic and quanti- 
tative descripLions of the manifest content of communication." 
In addition to tlie content analysir. techniques described earlier 
an introduction to content analyr.is may be found in Kerlinger 
(1973) v;hile more advanced treatments of the topic may be found 
in Budd jet ill- ( 1967 ) and Ilalste (1969). Grobman (1972) has 
provided a useful discussion of the uses of content analysis 
in f orma tive and summa tive evaluation althou^^h her discussion 
seems more oriented toward summa tive evaluation. 

Content analysis, hov;ever, does not lend itself easily 
to a consideration of the relationship among concepts in the 
subject matter. The learning structure analysis of Gagne (1970) 
is very useful in this regard. This technique, also described 
earlier in this i>ap-ej: under task analysis procedures, is 
relevant for the formative eva lua t ion • o f interim products. 
Gagne (1970) presents many examples of learning hierarchies 
and the technique seems to offer many advantages. 

The construction of learning hierarchies is quite time 
consuming, however, and the construction of a learning structure 
is no guarantee that it is "correct." In essence, the learning 
structure is a logical analysis of the objective and, as is^ 
well known, logic does not always simulate reality. Skills which 
are presumed to be subordinate to a particular objective may 
turn out not to be or the sequence of subconcepts may prove 
to be wrong. Learning hierarchies are, in essence, hypotheses 
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conccniinf!; the cont:c'nl^ hypotheses v;h:Lcli can oiiJ.y be confirmed 
empirically. The u s c f m .1 n c s of a particular loarninp hier- 
archy will depend upon hov; well it fits the reality of the 
situation. Many subject matters do not lend thomselves to 
hierarchical analysis. In other words, this, and systems similar 
to it, do not possess unlimited applicability but they should 
prove useful in many situations. 

The reader may, at this point, be wondering whether the 
collection of descriptive infornation is really necessary. 
Many vjould argue tliat all that really counts is hov; v;ell the 
product works, not what it consists of. The trouble with that 
arc;uraent is that not all products work, especially in first 
draft form. VJhen a product fails to perform as expected, 
explanations must be found. An adequate inventory of descriptive 
information v/ill assist greatly in locating the points at which 
the product needs revisions. The particular information 
collected will depend on many complex factors: cost, utility, 
past experience, etc. As such information is collected more 
often, the collection should become easier. Instruments such 
as a checklist will already be constructed, past content analysis 
systems already "debugged," etc. The reader is also referred 
to the CMAS, Eash, and Tyler and Klein analysis procedures 
referenced in the earlier paper by Sanders and Cunningham (1973) 
as well as procedures provided* by the Educational Product Infor- 
mation Exchange (EPIE) (1972). 
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The activiLy of critical appraifiial ciIgo refers to inspect- 
ing 'llu' product, iistjl r. Critical appraisal Jr. not tlio appraisal 
of the effects of a product upon people who arc using the 
product but upon people otlier tlian those directly involved in 
its use. Often the distinction between people *'usinj;" the 
material and those " a pp r a I r- i n the material is difficult to 
maintain for some techniques (e.g. individual student tryouts, 
to be discussed below), but the distinction has nevertheless 
proved useful . 

T h e techniques for collecting c r J. t i c a 1 c\]^p r a i s a 1 s o v e r 1 a p 
to a great extent v;ith the metliods of evaluating', objectives 
described earlier. Collections of opinions from experts of all 
sorts, teachers, parents, students, administrators, autliors, etc 
can be accomplished by means of questionnaires, checklists, 
interviews, panels, diaries, Q-sorts, the delphi technique, etc. 
The criteria against ''hich each of tliese populations can 
appraise the matfrx »s v/ill vary. Teachers will undoubtedly 
be concerned with such factors as congruence of content with 
their own biases or capabiJ it ies , practicality of the format, 
-mode, and/or requirements of the instruction, degree of inte- 
gration v;ith existing curricula, extent of teacher input, 
flexibility, and so on. Parents may be very concerned with 
the type of value system implied in the material, currency of 
content, orientation (i.e. to college bound or vocationally 
oriented students), sex or racial bias portrayed, and so on. 
Any or all of this information can bear upon tlie subsequent 
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revision of Ins l rue tii.'i 1 innl*orlcilr> especially v^/hcn extornal 
i n f or ma t i o n s u jj p o i- 1: r> I: li e critical a p p r a i s a 1 . 

One judgiiiLMi ta 1 data source v/liich ha?; not: been lapped in 
this chapt:cr vlrich deserves attenfion is the author. The 
dcfinit:ion of the? term autlior differs soine\7hat from medium to 
medium. In the ca£;e of the print media, text, or audio taper, 
the definition If. fairly easy. But with film or videotape the 
term author is probably closest in meaning; to director. The 
author is often not recognized as a source of revision infor- 
mation hut he is in fact: a raj or if not tlie major one especially 
at the early stages of the product. The author makes literally 
thousands of decisions when he embodies the content he intendr 
to teach in a suitable form', decisions concerning sequence, 
phrasing, orientation, value, difficulty level, and so on. 

When textbook authors write tlieJ.r prose they arc writing 
with a particular audience in mind, with a particular standard 
of difficulty and clarity. As the sentence is written, judg- 
ments are being made as to its adequacy in conveying intended 
raeaninp,s, the sophistication of the audience, the contribution 
of the sentence to the orderly development of the intent of 
the paragraphs, etc. If the sentence fails to meet these 
criteria it v;ill be rewritten until the author is satisfied. 
It should be obvious that estimating the number of these 
decislojis that the authors make as in the thousands is probcbly 
quite conservative. 

Authors, however, are often only diml.y aware of the 
decision process. Explicit standards are rare and, probably 
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n« a con«equoncc, con5;ir.t:oncy In decision making is less 
frequent uhan would bo. dcsiral^lc. Some profit: might; nccrue, 
therefore, by increasing author awareness of liis decisions, 
Lawson (1973) has constructed questionnaires and checklists 
which should prove useful in this ref^ard. In one of his 
ques tio iina i r cs , authors are querrled on whether specific 
learner ob j e c t iv es are prov J d ed , wh e t her entry b ehavior s are 
specified, v/h ether provision is rirade for learners to enter 
the product at points other than the beginning, v/hether the 
format and display are appropriate for the intended population, 
whether examples and illustrations used are likely to be of 
interest to the intended population, etc. The effects of such 
procedures upon autliors is urknown at this time and should be 
the object of future study. The v;illingness of authors (or 
more accurately what type of authors would bo willing or 
unwilling) to explicate their decision processes would be very 
Interesting to examine. 

It should be noted that the descriptive and critical 
appraisal techniques described in this section thus far on 
formative interim evaluation can be used at the formative 
product stage as v/ell. The difference is primarily one of 
the size of the "piece" or the closeness of an interim format 
that is being evaluated to the final product but thc^ principles 
involved are generally comparable from stage to stage. 
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Much useful cxti^rnnl in f o r ina t io n n t: Llic intcn-lin j^tngo 
can. bo BatJ^orccI by iKS.Ing t!io .'JciniC crit:(M":[o)j mocisuroij whJcli 
will bo u^tid at the formative product st'aj;c. If an achieve- 
ment test is carefully cons true tod to ni on sure tlie complete 
set of objectives of the in s t r u c t io n > there is no reason why, 
if the test has been carefully criterion referenced, that 
appropriate iteins from that test could not be used to evaluate 
"pieces** of the instruction designed to teach certain of the 
objectives. If, however, tlie criterion test does not measure 
every objective but merely samples from among many, then it 
would not be appropriate to use that test as an interim evalu- 
ative device. Although very desirable at the formative product 
stage it is manda tor y at the interim stage that some evaluative 
information be provided on every objective of the instructional 
product. 

The principles of construction of and the theoretical 
bases for external evaluation devices of many types should be 
familiar to the readers of this chapter and do not require 
reiteration here. That the evaluation of instructional 
products should emphasize the attainment of the particular 
objectives of tliat product (be c r i t er ion r e f er en c cd ) rather 
than Individual differences among students (be norm referencad^ 
is almost at the status of a truism these days. Likewise it 
is widely acknowledged that evaluation should be as direct 
and performance based as possible; that is, for example, if 
students are supposed to be able to correctly assnmhlc an 
automobile distributor after instruction, they should be 
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Cciilocl by {»lving IIicmm a ci Isas ruinil) led automobile distribution, 
nol by tcHt iny, wl(:h pnj'jcv and pcM)cil Lbeir hnowled[j,c of the 
functions of the d i I: r J b u t o r • Discussion of these issues can 
be found in any competent educational measurement text but we 
wish to rocomincnd especially the Handboo k o f Fornia t ive and 
Sumtua t iv e J'-va lu:i^t^:i^on hy Bloom, Hastings and Maddus (1971). 
We v;ill have more to 5;ay about more formal external evaluation 
devices and ])rocedures ia the last section of this chapter, 

At the interini sta^^e of product develo.^viCM; t , one should 
not limit his information gatlicring activity to hi.glily structured 
procedures, ^^uch useful inCorma tion can be gathered in informal 
types of ojierations. One which lias received increasing attention 
during recent years is variously called developmental testing 
(Harkel, 1967), individual student tryout (Scott & Yelon, 1969), 
and oral problem solving (Cunningham, in press). Essentially 
this technique consists of placing the author (or his agent) 
with one or more students as they use the materials. Ideally 
the student(s) will, by means of oral or v;ritten comments, 
help the autlior locate ambiguities, errors of sequence, and 
the like, and allov; the author to test his assumptions con- 
cerning the mental operations which will be employed by 
students using the material. The students are generally told 
to "think alouu" as they work through the materials, a proce- 
dvire which it is hoped will give the author insights into the 
students' thinking processes and into hov; well his materials 
have coordinated thenisclves with those processes. 
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U n f o i: t u t\ a t c 1 y , very 1 .1 1: t: 1. o ni p ;i. r i r. a J. k n o w 1 c d c? c x i s t <: on- 
c e r n i. ii g i n d i v i. cl u a 1 r> t u dual* L r y o u l « . U c y o n d a u u u p u b 1 i s h c d 
master's thesis by Kuback (1963) and some recent v;ork by Abedor 
(1972) discussed earlier in this chapter, very little research 
on the technique has been completed. The present state of tlie 
art is crude, consisting of n nuiabcr of i n sub s t an t i a t ed "tips*' 
as to hov; to carry off the procedures. And any inspection of 
the literature relevant to this topic quickly reveals the 
inconsistency and lack of agreeiTiont among those "tips.** Sorae 
recommend that high ability students be used, others recommend 
lov; ability. Some sources argue that students can only clean 
up semantic and syntactic errors while others insist that the 
student can make more substantive su^^gesLions concerning sequence, 
intended prerequisites, etc. Recommendations also vary vjith 
respect to preferred level of student incentive, author behavior 
in the tryout situation, number of cycles of tryout and revision, 
and on and on. At present few standard procedures can be 
recommended with confidence. Even the simplest of experiments 
con^parlnr; the quality of instructual products which have and 
have not used individual student tryouts as part of the develop- 
ment has yet, to our knowledge, to be completed. We hope that, 
in the next fe\; years, the research necessary to validate and 
refine these techniques will be completed* A more detailed 
discussion of the issues and con s J d cr a t J on s of individual 
student tryouts can be found in Markle (1967) and Scott and 
Yelon (1969 ) . 
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Systematic a s t; o s s men t of cont'oxt nt tliG xntnrlm Mtage is 
likely to be v.m s:; t c f u 1 j;;inci.> oiiJ. y p.iocos of the v>roduc t arc 
avnij.nl)lo« The impact of email pieces of a product on a parti- 
cular context is liko.ly to be un r op v csen t a t iv c of the Impact 
when the product as a vMioIe is in t e[.',ra tod i.ni:o a particular 
si tua Uion • 

The Gvaluator mui^t be av;are of intended contexts, however, 
to guide his clioice of students for individual student tryouts 
or small scale field tests or to guide in the choice of people 
to conduct critical appraisals. The systematic testing of 
context and the search for relationship between context and 
other inf ornin t ion about the product is best delayed, however, 
unti] the forvaative product stage. 

ADDITIONAL REFERENCES:^ Materials analysis guidelines (EPIE, 
1972), PERT (Cook, 1966), PPBS (McCullough, 1966), systems 
analysis (Clclaud and King, 1968; Kershaw and HcKean, 1959). 



TECHNIQUES AND PROCEDURES FOR 
FORMATIVE PRODUCT EVALUATION 



At this point in the product development sequence, a 
version of the complete product is produced. Rather than 
being discrete, this stage is continuous with evaluation of 
interim stages of the product. Most often the first evaluative 
information collected concerning the product as a v/hole Is 
the same Information collected at the interim stage and many 
of the same tccluiiquos are applicable. However, it is the 
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view lun*e that. l;lio nuijor llirur.t' oT the formal:ivo i>rocluct • 
evaluation u f I o r I: iUioiild 1)C' towaj'd the cvcMiLual tal) linhnut n I. 
of the? r ela t ioufpliip bcil\;ccMi contextual and other product 
c h a V a c t e.r i s t i c s . l^x t e r n a 1 validity b c c o \n o s c r u c i a 1 f o r for m a t :i. v c; 
product c V a 1 i i n t :L o n a c t i v 1 1 i c s . 

The majoi* lochnlqu:is for collecting Internal information 
( choc k] i s t s for d es cr i p 1 1 v c info r ma t ion , qu eo t i o nna ir es , 
interviews, etc. for critical appraisals) liave already been 
discussed and iwn e s s c t i a 1 ly the same for this sta[', . 

It is also possible v;h en co 11 oc t iu^ ex t crn al in f o nv.a t ion 
to use many of the same methods and procedures as v;orc used in 
the interim stage including individual student tryouts now 
vjith the complete product. The emphasis, however, nov? shifts 
to large scale tryout, vjhere the compi.ete ]iroduct is tried out 
under the circumstances in which it is supposed to operate. 
Although having an author hovering over a student is acceptable 
during d eve lo i^men ta 1 testing of an instructional program, it 
would not be acceptab].e in a field test of tlie product. 

An inventory of the possible measures wliich could be 
collected would be very large indeed, but Met-^^essel and 
Michael (1967) liave made a useful beginning. They list five 
major categories or what is here called external information: 

1. Indicators of status or change in Cognitive and . 
Affective Behaviors of f.tudents in terms of 

s tandnrd i:^.ed measures and scales . 

2. Indicators of status or change in Cognitive and 
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Afj'ocLivo BclinviovM of 5; i* ii d un I: s by In formal or 
soincii fornuil t c a c li or- ni ad (i 1 n Ji t r u iulmi t; s or doviccn. 

3. Indicators of' 5; talus or chanj^; 0 in indent bcliavlorn 
othor tlian tliosa mcnsurod by l:cst.s, J n vcmi t o r i c a , 
and ob^sorva t I on scalos in rola tion to iho task o f 
o va lua I in nb j cj c t i v c s of school pro ^^r ams . 

A. Tndicaloi's of status or clian[>,c in Cognitive and 
Affective 1) eh a V i o r s; of 1 0 a c lie r s and o t ii e )• i3 cho ol 
personnel in relation to the evaluation of school 
p 1" o r a in s . 

5* Indicator?', of community behaviors in relation to 
the evalua tion of achool programs . 
Under thes;c five hcadingfj arc listed 'many particular Informati.on 
•sources including; unobtrunive sources. The stratc[^y ve vjish 
to emphaiiize here is the use of muH-iplc criterion measures in 
which all criterion measures arc recogni?.ed as fallible and in 
need of collaboration by other methods whose fallibilities are 
likely to be different from tlie first measure* An attitude 
scale wlrich prrrports to measure attitude toward a subject would 
be more credible if it could be shown to correlate highly with 
some unobtrusive measure like the proportion of book^ checked 
out of tlie librai-y on that subject or with a classroom obser- 
vation scliedule v/liicl) demonstrates a high proportion of activi- 
ties related to the subject during; free periods* 

Duo t o s p a c o 1 i m i t a tion, we c a n n o t p o s i 1) 1 y d i s c u s s a 1 1 
of the many pos5;ible sources of external in f o naa t ion . Conse- 
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qutMiLjy linvc^ clioMMi lo disciuif; iIiomc pt^ohlrniM found l;o be 
p i\ V I. i-C u J *i r 1 y np pii i nil i ii j; r vr i J 1 p ro d u c I d c v c 1 o n I of i v r I ?i . 
At tUo on I of thJj; .'.oclion vu' will ro f cr iMU'cii for other 

m a J o r mo l h o d i; f or a U» cm* 1 u \\ v. \ I o r u a I 1 n f o r m a 1 1 o u • 

Hy far i!ie woi.l froqvionily nf)ur,hl after outcomes from 
jnslriiction are ylmhMit cop,nltivo oulcoincH, cnpecially hlj^her 
order coj'jiLiive ouLooiiios iuich at; concept lonrninf; or problem 
solvJtij;. Yet it often is I be case tb«Tt Lbe criterion mcaouros 
of these objertlvei; do not allow the inference that hlp»lier 
order ouieoiiie^ liave occurred. Consider the folJowinj; paraj; raph 
wliicli mij;l)l be takcMi from an introductory mooRu rc'men t text and 
fiOiac potential test items. 

The mean is the avcrap.o score of a set of 
scores and iy computed by ilivJdinj> the sum of all 
the sc or ef3 ob t a iucd on the test by the numb or of 
the scores. If ten ntndentii r.covo 1, 3, A, A, 5, 
5 , 6 , 7 , 7 , and 7 rospoc tivoly , then the mean would 
be A9 -t by 10 or A.y, The mean is the most frequently 
used measure of central tcnidoncy. 

1. The is the most frequently u«ed 

measure of central tendency. 

2. Define mean. 

3. What is the mi^nn of the fol]o\;lnf; set of r.core.s? 
1, 3. A, A, 5, 5, 6, 7, 7, 7 

a • A 
b . A . 9 
c • 5 • 1 



A, Thu nu*asuro llial Js iiiosL o f i imi used t;o cl o r. i: r J be 

t h o ri V 0 V a y, i • s c o r c o f a cl I f ; t r .1 Im i t i o n I f li c 

5. In your v.'onlj;, dcfiiiC! t:lio. Mcnn, 



6, CoM)nilc I ho ncan o L* Lliis set: of scores, 
12, 19, 15, 30, 57 
a • 2 6,6 
b . ]. 9 
c. 21.5 

Hopc»fully you will have noted that iloiny 1-3 deiiumd nothinj;^ 
noro than verba tim recall or rcjcognitloii on the part* of the 
student. The r>ludonl. need only remeiubcr the form of the infor- 
m a t :L o n a .s It was stated in i n r r r u c t i o n s i nee t h e o r d i n ^; of or 
cxain])les used in the test xlefiis docs not differ 5;u bs t an t J a II y 
from the wording and examples used in instruction. Students 
answorins quor.tlors 1-3 correctly co\i ] d have an understanding, 
of the concept of mean as expressed in the brief passage but 
the items used to tost the concept do not unamb 1 ^;uous ly allov; 
that inference. Items /^-G adequately test the hipjier order 
objectives in that students probably could not answer thor. e 
items on tlie basis of verbatim recall or recognition alone. 
Key sentences from instruction have been paraphrased, examples 
have been changed. 

As obv i ous as t.h 1.*; point may seem to some , it is appa r e n t ly 
not* obvMuis t.o many of iho fornu'itivc^ and summative evaluators, 
l^erhaps under tl)(^ influence of lha 
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po J' f orinnnc (• con t r nc I i d ^; fad, n.ll too of I on n vory t'rlv.lal sort: 
of *' t oao li I 1) for l.lu' t.(.'5;t:*' can l)o soon In many pi-'oduct evalu-- 
citions. Su cli information can he f^rostily iiiiy load In^^ concern ing 
Ulio actual ]ovel of a 1 1 alniuen I of pai'tlcular outcomes. Parti- 
cularly lucjc! d i .sc ur, j; j o n s of t lie so issues can be found in 
Hormuth (1969) and Anderson (1972). 

A second problem area is in the procedures for collecting 
Information about effects of the product. Data concerning the 
effects of a product are not collected at random but, rather, 
according; to some pl^^ii which will allo\.' an assessment of the 
effects of the product in relation to some other state of 
affairs. It is at this stage of product development where 
experimental and qu a s i- exp er imcn t a 1 designs are useful. The 
standard reference on this topic cont:lnues to be Campbell and 
Stanley (1963), a truly outstanding summary of the relevant 
consideration in c:;pcrxmcntal designs. This chapter s>iould he 
part of the arsenal of every formative evaluator. 

Th( choice of design for a formative product evaluation is 
a complicated decision depending upon a number of considerations: 
cost, utility, practicality, tolerance for certain forms of 
invalidity, extent of r>enerali^:abili ty desired, and so forth. 
Campbell and Stanley (1963) have discussed the r.iajor consider- 
ations in tlie choice of a design: internal and external validity, 
or, alternatively, r epl icab il 1 1 y and gencraliiiability . The 
evaluator needs to be concerned with r op 1 j c ab i 1 i t y in that if 
the effect of his product cannot be reallably established, then, 
of course, decisions about how to make the product better are 
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meaning;] csH • The forr.intxve ovaluntor inuiil: also be runisitlvc 
to the extent: ami typo oC j; cn ax"n 1 i j'.ab il i t y of his product, l!o. 
may not be inturcstod in iiunkinj» generalisations from his evalu- 
ation to other products or otlicr contexts tlian the intended one 
but vrithin the intendiul contexts he has to take steps to ensure 
general ijiabil ity . Campbell and Stanley (] 963) list eight 
potential sources of Internal invalidity and four potential 
souj'ces of external invalidity. Each design discussed in their 
chapter is evaluated against these threats to validity at the 
consumer of these designs is able therefore to choose those 
designs v;hich in in iin i :i e s threats of most concern. 

VJithout doubt, the most frequently used design in product 
evaluations is tlie single group pre-test post-test design. In 
this quasi experimental design a single group of students is 
first tested to determine how much of the terminal behavior 
they possess, then are administered the product, then tested 
again, often vith the same test. If learning gains are demon- 
strated, the product developer will conclude he has a successful 
product. The problem witli such a design is that it allows so 
many other plausible rival explanations for the observed result: 
other events occuring between the first and second testing may 
have caused the results, the pre-test alone may have influenced 
the post-test, shifts in standards and scoring pre-test and 
post-test could occur, just to mention a few. Markle (1967) 
has pointed out that improvements in post-test performance can 
often be shown to be due to an increased familiarity with 
terminology used in the product rather than any new learning. 
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In Gum, l\)it\ clesl|;n tiocj; not bavo a gronl: deal t:o offer oxc cyt 
that il: .1g ]>rc}bab.l.y bottar than notbin^;. As the jonJ:.,y type of 
cvaluaLicMi for tlie product it is inadequate but as a firnt step 
in a more o X a borate v. t: of p r o c o d 1 1 res, i. t can serve a u s o f u 1 
function^ When it i^ the only posBlblc dcsi^»n, care should be. 
taken to investigate as many as possible of the potential 
sources of invalidity specified by Campbell and Stanley (1963). 
More fruitful dcsi^^ns have been discussed by Glass in Wort hen 
and Sanders (1973) . 

VJith vep,nrd to contextual information, developers typically 
liave an average student, a particular average classroom setting 
in mind when they construct a product. It is the function of 
the formative evalnator to identify and malce explicit those 
assumptions and then to provide a context (if one exists) for 
the field test. This description implies a two stage process; 
the identification or. intended contexts and then the testing of 
products within specified contexts. Testing may force n 1 fi- 
cation of the intended context or of the product so that it 
better fits a more realistic set of context variables. There 
is thus no mystery to the collection of contextual information 
in that it involves the use of instruments already discussed 
in this chapter. Questionnaires or interviews v;lth the author 
could be used to identify intended contexts on such variables 
as entering behaviors, student attitudes, socioeconomic status, 
student interest, teacher experience, teaching stylo or person- 
ality, etc. The intended curricular context for the product 
Including the type(s) of concurrent course work, availability 
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of Itifi Iriic t:u r.-il axclii, :nul the like should alijO bo assessed. 
Other varinhlos aro d xjj cvi s cd in Cunningham (in press) and 
S a n d e r s a n d (1 u n n in h am (19 7 3) • 

The identification of actual contexts will center upon 
the intended contextual variables but the evaluator should be 
aware of and sciisiitive to other context variables which inight 
conceivably influence tlie outcome of the field test. Systeniatic 
obnervatlon and survey instruments similar to those used for needs 
asscssinont could be used to collect this information* 

It was stated earlier that the proper focus of formative 
product evaluation is on the establishment of relationships 
between context, other information about the product. It wps also 
stated earlier tlia t the purpose of formative evaluation is to 
provide information of use to the developer of the product con- 
cerning potential revisions. These revisions will be most 
efficiently and effectively made if all of the information 
discussed in tliis chapter are available to the evaluator. If, 
for instance, it is demonstrated that students have failed to 
master a particular objective, the formative evaluator must 
find cut why and determ.ine what to do about it. He should at 
tha t po in t b egin to hy po th es ize various patterns of relation- 
ship among all of the in for in at ion already collected. Were 
student entry behaviors over'-estimated (context)? Was the 
read-ability level of the text at that point too great 
(descriptive)? Did subject matter experts predict difficulty 
with tl)Ose concepts and, if so, why (critical appraisal)? 
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Any OIK? or combination of tlicse c. on s 1 ilc» r a I io n s could conco. ivab 'l.y 
shed li^;hl: upon t:hc p a v (: i im ] a r dt'ficioncy identified and 
perhaps; imply the slci)s whicli should be talion to rcMiicdy the 
situation. In tho courye of this procedure, relationships arc 
Identified whicli mij;;ht have some dc-gree of g en e r a 1 1 / a b i 1 i ty to 
other problom.s v/ithln the same product or perhaps even with 
other products as v/ell. 

The potential usefulness of such a focus has been discussed 
already in Sanders and Cunuingliam (19 73) with respect to a 
field test conducted by Anderson (].969 ) in V7hich he found that 
a discrepancy between an intended context factor and an actual 
one could account for some disturbd.ng external information. 
Other examples could be cited. A student of the second author 
V7as at a loss to explain \7hy students didn't seem to profit 
from being provided with knowledge of results in her self- 
instructional program. A cursory glance at the internal 
characteristics revealed a great proportion of formal prompts, 
so many in. fact that the program was too easy. Knowledge of 
results after each frame v;as simply not needed since more tlian 
sufficient information about the correct ansv/er was contained 
in the frame itself. 

To conclude this section, v/e raight reiterate that we 
recommended the collection of multiple measure of many types 
of in forma t ion and the search for relationships among these 
data. VJe are not so naive, however, to expect that formative 
cvaluators have unlimited time and r esour ees ava 11 ah If? to them 
to pursue all of the recommendations which liave been made. 
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VHunt. 1b pvopofietl ir. ;u\ lilcal, a j^onl to strive for rnthor than 
a do p m a I: i c set of p r p s c r i p t i o n s . 

ADDrnONAl. REV1:ri-;;CKS : cost annlynis (Trcst and Turvey, 1965 
Tanner, 1<J71; Fisher, 1971),. GFE (Scriven, 1972), ATI (Cronbach 
and Snow, 1969). 
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Tlui first author In ciirrcnL.ly n S tailor Rorioarcli Af.isocIat:c 
at the N o r t li v; e « t R o\;\ i o 11 a 1 K d u c a t i o n n 1 L a b o r a L o r y . 

Inforinatlon about yaniplJu^^ plans used by NAKl^ and ISll is 
cent a i n e d 1 n s e v c r a 1 d o c u ni e n t: s p u 1) 1 i s !i e d b y t hose 
institutions. The reader is directed to the list 
of references at the end of tliis chapter for 
references to documents which contain brief summaries 
of the s a ?.'] p 1 i n g plans. 

A computer program useful for analyzing Q--sort data has 
been prepared by B a u m an ( ]. 9 6 9 ) . 

This example was taken from Gagne (1970 ) • 

"ADDITIONAL REFERENCES** provided in this chapter comprise 
works that have not been previously referenced in 
Sanders and Cunningham (1973) or selected for 
elaboration here. References contained in the 
Sanders and Cunningliam (1973) article have not, 
for the most part, been repeated in this chapter. 
Hence, the reader may v;ish to combine the two 
documents for a more complete treatment of methods 
for formative evaluation in product development. 

The Lav;son (1973) article has been v;ritten within the 
Sanders and Cunningh»am (1973) framework. 

This example was adapted from an illustration provided 
by Berelson (1952) . 
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